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ABSTRACT 


Simulation,  once  the  rarefied  domain  of  engineers,  is  increasingly  subject  to  scrutiny  and 
management.  Once  considered  something  of  an  oracle,  it  is  now  “business  as  usual”  for  many 
managers.  As  such,  it  must  compete  favorably  for  attention  and  funding  with  the  very  end- 
products  whose  development  it  facilitates.  In  a  world  of  product  development  increasingly 
dominated  by  business-school  and  marketing  managers,  we  must  be  able  to  articulate  the 
quantifiable  benefits  of  our  efforts  in  both  magnitude  and  duration.  This  report  presents  a 
“checklist”  for  Return  On  Investment  (ROI)  in  four  classes  of  simulation:  “Constructive,” 
“Virtual,”  “Live,”  and  “Smart.”  Each  is  distinguished  by  its  degree  of  simulation  (versus  “real”) 
of  equipment  (end  products),  people  (end  users),  and  the  end-user/product  environment.  By 
establishing  clear,  reasonable  expectations  for  ROI  in  simulation  -  across  the  spectrum  of 
modeling  and  simulation  disciplines,  and  throughout  the  product  life  span  -  we  establish  baseline 
cost  estimation  methods  that  can  survive  internal  and  external  economic  competition. 


FOREWORD 


This  special  report  combines  a  paper  submitted  to  the  Society  for  Computer  Simulation 
(SCS)  as  part  of  the  Summer  Computer  Simulation  Conference  (SCSC)  2001  with  its 
accompanying  presentation  materials.  The  charts,  illustrations,  and  notes  are  intended  to 
encourage  listeners  to  delve  further  into  the  details  found  in  the  paper  rather  than  present  findings 
in  “the  paper”  as  such.  Consequently,  the  reader  will  find  that  reading  of  both  sections  of  this 
report  will  comprise  a  much  fuller  story  than  either  section  taken  alone. 
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I.  BACKGROUND 


Those  of  us  who  imprinted  in  the  Modeling  And  Simulation  (M&S)  community  remember 
the  good  old  days  when  the  word  “simulation”  automatically  opened  all  sorts  of  doors.  Software 
was  the  oracle,  computers  were  its  temple,  and  we  were  its  priests.  As  the  complexity,  cost,  and 
visibility  of  test  and  simulation  has  increased,  the  invitation  to  thoughtfully  examine  their  value 
through  the  cold  lens  of  Return  On  Investment  (ROI)  has  sounded  louder  and  louder  [1,2]. 

We  generally  make  simulation  technology  investments  well  in  advance  of  formal  Project 
Manager  (PM)  requirements;  ideally,  they  are  matured  along  with  other  key  enabling 
technologies.  If  designed  with  adaptable  architectures,  they  can  be  used  and  re-used  many  times 
by  different  PMs.  A  business  case  for  M&S,  then,  would  define  expectations  of  return  in  specific 
business  areas  over  specific  phases  of  a  program’s  life.  As  validated  case  studies  are  scarce,  this 
report  attempts  only  to  assert  some  order  and  expectations  regarding  the  ROI  of  M&S,  without 
attempting  to  defend  their  exact  magnitudes. 

II.  FRAMEWORK 

Rather  than  address  every  species  of  simulation,  Figure  1  illustrates  M&S  in  a  way  tutorial 
to  this  discussion.  A  Three-Dimensional  (3-D)  framework  offers  a  useful  description  of  this 
domain.  The  first  dimensional  axis  is  the  “equipment”,  or  the  end  product.  The  second  is  the 
equipment’s  operator  -  the  human  customer,  end-user,  or  beneficiary.  The  third  is  the 
environment  in  which  the  end  product  and  its  operator  must  operate.  Each  axis  represents  a 
continuum  (illustrated  for  discussion’s  sake  as  discrete  combinations)  from  least  “reality”  to 
greatest.  Frequently,  the  cheapest  corner  in  this  domain  is  found  in  the  area  of  constructive 
simulations.  By  contrast,  the  most  expensive  comer  of  this  cube  is  the  gold  standard  for  most 
PMs  -  Operational  Test. 
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Fundamentally,  M&S  and  Test  exist  for  the  purpose  of  evaluation,  and  evaluation  exists  in 
order  to  enable  and  support  decisions;  e.g.  what  to  do,  which  to  buy,  etc.  Rational  economic 
behavior  would  be  to  make  effective,  efficient  decisions.  Accordingly,  we  consider  minimization 
of  time  spent  in  test,  and  reduction  of  risk  in  getting  to  it,  a  laudable  goal  technically  and 
financially. 

More  importantly,  by  moving  to  a  monetized  system  of  valuation,  we  invite  common- 
language  discussion  of  the  need  for  simulation  vis-a-vis  the  market,  the  product  itself,  and  the 
array  of  supporting  processes.  Most  important  of  all,  in  a  world  of  customers  increasingly 
populated  by  Masters  of  Business  Administration  (MBAs)  instead  of  engineers,  we  will  choose 
to  speak  their  language  rather  than  insist  upon  their  learning  our  own. 

III.  UNDERLYING  COST  DRIVERS 

The  framework  above  suggests  a  three-lobed  question  that  should  be  asked  early  in 
simulation  definition  and  development:  “To  what  extent  must  we  simulate  people,  equipment, 
and  environment  in  order  to  make  our  next  decision?”  Underlying  cost  drivers  emerged  as  this 
cost  model  was  developed.  Many  others  may  come  to  mind;  two  are  addressed  here. 

A.  Availability 

Cost  of  evaluation  is  frequently  proportional  to  fidelity  and  complexity,  as  asserted  in 
Figure  1.  For  example,  a  statistical  (constructive)  model  of  missile  reliability  as  a  function  of 
age,  storage  conditions,  and  temperature  is  far  less  “real”  than  a  reliability  firing  test  program  - 
but  if  validated,  is  far  simpler  and  more  affordable.  Similarly,  Developmental  Test  (DT)  and 
Operational  Test  (OT)  under  outdoor  conditions,  offers  “the  greatest  realism  short  of  war,”  but 
places  the  test  at  the  mercy  of  the  outdoors.  People  are  notoriously  difficult  to  “model,”  and 
many  models  simply  abstract  some  initial  operator  state  for  their  systems. 

Underlying  this  cost  increase  is  the  pernicious  tendency  of  test  availability  (the  fraction 
of  time  in  which  all  components  work  at  the  same  time,  correctly,  together  with  all  necessary 
measurement  systems)  to  go  down  with  complexity.  Figure  2  illustrates  “test  availability”  as  the 
probabilistic  combination  of  some  dozens  of  parts,  each  with  some  assumed  “part  availability.” 


Total  Parts  Count  (includes  system,  subsystems,  people, 
instrumentation,  and  evironmental  variables) 


Figure  2.  Inverse  Exponential  Relationship  Between  Test  Availability  and  Complexity 
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It  is  immediately  obvious  that  “more  parts”  means  “more  things  to  go  wrong”  (per 
Murphy’s  Law)  -  and  that  compensating  for  breakage  in  complex  test  architectures  means  the 
expenditure  of  a  great  deal  of  money  not  to  improve  fidelity  or  thoroughness,  but  simply  to 
execute  the  test.  A  useful  rule  regarding  end-item,  support,  and  instrumentation  equipment  for  a 
test  is:  “If  it’s  worth  taking  one  of  something,  it’s  worth  taking  two. . .  at  least”.  M&S 
consistently  offers  cost  savings  rooted  in  its  far-higher  availability  when  compared  to  field  test. 

B.  Descriptive  vs.  Prescriptive  M&S 

Descriptive  models  depict  the  behavior  or  properties  of  existing  systems,  while 
prescriptive  models  convey  the  expected  behavior  or  properties  of  a  proposed  system.  [3]  The 
differences  for  cost  estimation  and  business  case  development  are  frequently  obscured, 
intentionally  or  unintentionally,  but  are  profound. 

Costs  for  use  of  descriptive  models  are  composed  of  design,  implementation, 
Verification,  Validation,  And  Accreditation  (VV&A),  and  employment.  All  but  actual 
employment  costs  can  be  avoided  entirely  via  reuse  -  hence  decades  of  attempts  to  legislate, 
regulate,  demand,  compel,  and  beg  for  it.  By  now,  if  reuse  were  an  economically  rational 
behavior,  we  should  expect  libraries  full  of  fully  accredited  models  of  everything  that  opens  and 
shuts.  Simulation  developers  should  be  able  to  select  from  a  large  catalog  of  models,  and  make 
“make  vs.  buy”  decisions  with  regularity.  With  a  few  notable  exceptions,  this  has  not  happened. 
By  blandly  asserting  that  we  should  build  reusable  M&S  components  for  the  common  good,  we 
shout  against  the  tide  of  economic  necessity,  which  is  to  first  advance  our  own  interests.  As  we 
assert  cost’s  role,  however,  imprecations  from  above  to  reuse  M&S  can  be  personalized  and 
localized,  saving  money  for  your  own  program. 

Prescriptive  models  are  easier  and  cheaper  in  that  they  cannot  be  disproven  by  reality. 
However,  costs  for  prescriptive  models  of  systems  that  do  not  yet  exist  are  notorious  for 
hemorrhaging  large  amounts  of  cash  with  very  little  tangible  return.  By  definition,  they  model 
things  that  may  be  years  in  the  future;  the  customer  is  rarely  satisfied  with  the  video  game 
because  it  creates  in  him  a  strong  desire  for  the  real  thing  -  now  that  he  knows  what  that  will 
look  like.  But  the  major  cost  estimation  problem  with  prescriptive  models  is  the  large  number  of 
cooks  stirring  the  pot,  many  of  whom  are  transitory.  The  result  is  a  rubber  baseline,  from  which 
no  one’s  wallet  is  safe. 

Rational  economic  behavior  for  the  product  developer  is  to  gain  decision-making 
information  effectively  and  efficiently,  as  discussed  above.  There  are  strong  economic  incentives 
to  minimize  descriptive  M&S  costs  (in  whatever  way,  be  it  reuse  or  simply  choosing  an 
appropriate  level  of  fidelity),  and  focus  prescriptive  costs  on  the  modeling  of  the  new  product 
alone.  For  example,  a  manager  for  a  new  sensor  would  tend  to  buy  atmospheric  propagation  and 
target  models  off  the  shelf,  while  focusing  prescriptive  funding  on  the  sensor  itself. 
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IV.  CASH  FLOW  MODELING  FOR  M&S 


Discounted  cash  flows  (Fig.  3)  should  be  developed  for  quantifiable  [2]  M&S  investments, 
using  sensible  ROI  requirements,  planning  horizons,  and  consideration  of  the  cost  baseline 
(frequently,  test).  Methodical  cross-checking  of  a  requirements  Test  Verification  Matrix  (TVM) 
against  the  project’s  baseline  cost  estimate  can  identify  additional,  perhaps-unbudgeted  M&S 
requirements.  If  the  payback  is  negative  (frequently  the  case  for  short-term  M&S  programs)  then 
unquantifiable  [2]  paybacks  should  be  explicitly  listed.  This  offers  a  clear  articulation  of 
quantifiable  payoffs  and  the  collective  cost  to  the  program  of  “buying”  the  unquantifiable 
benefits. 


Figure  3.  Simplified  Cash  Flows  for  a  15-month  M&S  Effort  with  200  percent  ROI 

M&S  as  a  business  discipline  is  often  outside  of  our  customers’  training.  For  example, 
would-be  PMs  at  the  Defense  Systems  Management  College  (DSMC)  are  familiarized  with  cost 
estimation,  systems  engineering,  earned  value,  and  test.  However,  the  utility  of  M&S  in  the 
intersection  of  those  disciplines  is  not  currently  part  of  their  curriculum.  Our  business  model 
must  communicate  to  them  the  periodic  and  cumulative  effects  of  our  efforts,  both  forecasted  and 
actual.  Similarly,  in  most  industries  we  have  business  people  and  engineers,  but  no  clear 
educational  or  professional  processes  for  the  business  of  M&S  as  it  applies  to  the  entire 
enterprise.  [1]  Cost  Estimating  Relationships  (CERs)  are  elusive  in  this  environment,  but  ripe 
for  assertion  as  we  engage  our  customers  in  their  language. 
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V.  ROI  OF  CONSTRUCTIVE  SIMULATION 

(simulated  people,  equipment,  and  environments) 

Payback  of  constructive  modeling  is  frequently  either  unquantifiable,  or  only  quantifiable  in 
very  gross  terms;  e.g.,  “rejected  3  other  system  architectures  as  impracticable,  with  a  cost 
avoidance  of  $10M.”  A  model  familiar  to  all  PMs  is  a  program  cost  model  -  every  budget, 
schedule,  and  network  we’ve  ever  seen  was  simply  a  representation  of  a  program.  Looking  at  old 
budgets  and  schedules,  we  can  easily  conclude  that  these  models  frequently  suffer  from 
inaccuracies.  Imagine  a  PM  without  his  schedule  chart,  whose  worth  is  entirely  unquantifiable; 
yet,  this  most  important  of  all  models  is  very  often  simply  assumed  to  exist  in  valid  form. 

Trade  studies  have  the  potential  to  save  the  most  money  (albeit  unquantifiably),  the  earliest, 
during  the  concept  and  technology  development  phase  of  a  program.  By  helping  to  make  smart 
decisions  (or  at  least  avoiding  really  dumb  ones)  that  commit  the  “90  percent  cost”  part  of  a 
system-to-be  to  the  right  path,  they  avoid  false  starts  and  save  much  time  and  money.  While 
generally  regarded  as  valuable,  when  misused  they  can  easily  contribute  to  “paralysis  by 
analysis,”  subtract  value  from  a  program,  and  squander  M&S’s  credibility  for  the  future. 

Simulation  of  processes  throughout  many  disciplines  allows  managers  to  try  alternatives 
before  they  buy  them.  These  can  be  as  physically  realizable  as  production  layout  and  material 
flow  models,  or  as  ethereal  (but  no  less  real)  as  business  processes.  Process  simulations  find 
lifelong  application  in  value  engineering  and  process  streamlining  applications;  cost  savings  from 
these  are  particularly  quantifiable  as  they  have  the  status  quo  from  which  to  baseline. 

Trade  studies  are  actually  an  early,  special  case  of  system  and  subsystem  design  and 
performance-prediction  models  of  all  types,  including  production  process  simulations.  These 
“engineering  simulations”  may  and  should  address  key  performance  parameters  such  as  vehicle 
range,  maneuver  envelope,  weight,  sensor  range,  etc.  Again,  most  of  their  payoffs  will  be 
unquantifiable,  although  modelers  should  be  encouraged  and  challenged  to  quantify  savings  as 
often  as  possible. 

Computer-Aided  Design  (CAD)  is  now  so  ubiquitous  that  we  tend  to  take  it  for  granted  in 
design,  but  investment  in  a  well-conceived  CAD  model  has  far-reaching  payoffs.  CAD  allows 
not  just  design,  as  the  name  implies,  but  analysis  of  all  mechanical  and  (recently) 
thermal/environmental  effects  on  systems.  Its  sister  technology,  stereo  lithography,  enables 
manipulative  visualization,  revealing  hidden  problems  and  opportunities.  Expect  a  payoff  of 
CAD  in  Computer  Aided  Manufacturing  (CAM),  as  the  now-analyzed  design  is  transferred 
digitally  to  computer-driven  machine  tools  for  fabrication.  Beware  of  cost-killers  such  as  file 
transfer  incompatibilities. 

Prescriptive  CAD  models  used  with  descriptive  fluid  flow  modeling  are  the  key  elements  of 
the  versatile  field  of  Computational  Fluid  Dynamics  (CFD).  Models  as  diverse  as  jet  plumes, 
propellant  flow,  airflow,  and  even  carotid  artery  flow  can  be  developed  to  analyze  mechanical, 
thermal,  flowfield,  and  aerodynamic  effects,  reducing  requirements  for  subscale,  wave  tank,  or 
wind  tunnel  modeling.  Cost  savings  are  found  in  reduced  reliance  on  such  facilities. 
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Well-done  CAD  is  foundational  to  survivability  models  of  our  tactical  systems.  These 
models  can  and  should  also  be  expected  to  reduce  the  cost  of  field-testing.  [4]  Target/system 
signature  and  environmental  effects  models,  against  innumerable  sensors,  pay  off  in  range 
avoidance.  Signatures  and  effects  can  be  re-used  infinitely,  versus  chancy  and  variable 
signatures  from  the  very  few  representative  “real”  targets.  One  model  in  current  use,  the 
Battlefield  Environmental  Weapons  Systems  Simulation  (BEWSS),  incorporates  models  whose 
“memory”  extends  back  to  the  mid-70s.  The  Virtual  Targets  Center  (a  joint  venture  of  AMREC 
and  the  Army  Simulation,  Training,  and  Instrumentation  Command  Targets  Management  Office) 
is  an  online  repository  of  validated  models,  which  is  accessible  to  all  the  services  and  industry. 

[5]  This  is  a  kind  of  memory  that  improves  with  age  and  is  shared  with  an  entire  community  for 
re-use.  Models  such  as  these  make  examination  of  multi-dimensional  target  effects  practicable  in 
that  the  marginal  cost  per  combination  of  target  and  weapon  variables  is  very  low  when 
compared  to  DT  and  OT. 

Similarly,  CAD  is  foundational  to  weapons  effects,  survivability,  and  lethality  models. 

These  models  pay  off  quantifiably  by  replacing  real  targets  (destructive  test)  with  simulated  ones 
(nondestructive  test;  subscale  or  digital  reuse).  Digital  targets  can  be  “blown  up”  millions  of 
times  while  “real”  targets  tend  to  be  totally  destroyed  (or  at  least  rendered  “invalid”)  in  a  few 
shots.  Typically,  these  models  are  used  extensively  in  early  design.  (In  missile  systems,  they 
sometimes  actually  migrate  into  subsystem  requirements  and  tactical  hardware  algorithms  for 
target  aimpoint  selection-  another  quantifiable  payoff.)  Sound  test  and  validation  is  essential. 
Savings  are  simple  to  compute  as  weapon,  target,  instrumentation,  and  range  costs  mount  up 
quickly.  Lethality  models  are  also  a  form  of  memory,  incorporating  findings  from  many  previous 
tests  -  thereby  leveraging  previous  customers’  investments  and  improving  ROI.  Similarly  to 
CAD,  circuit  simulation  is  now  such  an  integral  part  of  board  design  that  it  is  difficult  to  separate 
as  a  “simulation”  discipline.  Current  circuit  design/simulation  packages  help  with  circuit  design 
and  board  layout,  then  output  a  file  that  we  can  take  to  any  board  maker  in  town  and  have 
hardware  in  hand  in  a  few  weeks.  However,  when  properly  planned  and  leveraged,  it  can  pave 
the  way  for  fabrication  of  tactical  equipment,  test  equipment,  and  “simulation”  variants  or 
modules  sharing  a  common  design  process  and  family.  Quantifying  payoff  of  the  shortened  cycle 
time  is  not  too  difficult,  given  that  each  board  simulation  can  be  integrated  with  others  to  achieve 
“black  box”  and  “major  subsystem”  levels  of  integration  without  a  drop  of  solder. 

One  highly  quantifiable  area  of  constructive  modeling  is  in  predicting  service  life  and 
reliability.  These  models  pay  off  in  system  failure/crash  avoidance  (suspend  equipment  from 
service),  spares  cost  savings  (buy  adequate  but  not  excessive  quantities  of  spares),  and 
replacement  cost  savings  (safe/reliable  service  life  and  defer  replacement).  Combined  with 
validation-driven  sampling,  monitoring,  and  test  programs,  these  models  are  indispensable  to 
efficient,  effective  management  of  fleets  and  stockpiles  of  materiel. 
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Table  1.  Duration  and  Quantifiability  of  Constructive  Simulation  ROI 
Throughout  a  System’s  Life 


Quantifiability  of  Simulation  ROI  Throughout  Phases  of  a  System's  Life 


Constructive  Simulation  Application 

Concept/ 
Tech.  Dev. 

Sys  Design/ 
Demo 

Production/ 

Deployment 

Support/Operations 

Market  +  Cost/program/schedule 

Low  ($) 

Low  ($) 

Low  ($) 

Low  ($) 

Trade/concept  Studies 

Low  (t,$) 

Low  (t,$) 

Process  Simulation 

Med  (t,$) 

Med  (t,$) 

Med  (t,$) 

Med  (t,$) 

Design/Performance  Prediction 

Med  (t,$) 

Med  (t,$) 

Med  (t,$) 

Med  (t,$) 

Mechanical  CAD/simulation 

Med  (t,$) 

High  (t,$) 

High  (t,$) 

High  (t,$) 

Computational  Fluid  Dynamics  (CFD) 

Med  (t,$) 

High  (t,$) 

High  (t,$) 

High  (t,$) 

Target  &  System  Signature/Environ. 

Med  (t,$) 

High  (t,$) 

High  (t,$) 

High  (t,$) 

Weapons  Effects,  Survivability,  Lethality 

Med  (t,$) 

High  (t,$) 

High  (t,$) 

High  (t,$) 

Circuit  CAD/simulation 

High  (t,$) 

High  (t,$) 

High  (t,$) 

Service  Life/Reliability  Prediction 

High  (t,$) 

VI.  ROI  OF  VIRTUAL  SIMULATION 

(real  people,  simulated  equipment  and  environments) 

The  word  “virtual”  has  taken  a  beating  from  overuse  and  overgeneralization.  Here,  we  use 
it  in  a  limited  sense;  somewhat  facetiously,  virtual  simulation  is  about  big  video  games  in  which 
real  people  operate  simulated  systems  in  (usually)  simulated  environments.  Anyone  who  has 
ever  “flown”  a  helicopter  in  a  battle  simulation  quickly  gains  a  visceral  appreciation  of  just  how 
lethal  ground  forces  can  be  -  and  sometimes  gets  airsick  in  nap-of-the-earth  flight. 

Virtual  simulation,  taking  CAD  modeling  to  immersive,  synthetic  environment  [3]  levels 
for  its  users,  can  communicate  the  “art  of  the  possible”  to  a  customer  community  primarily 
focused  on  problems  in  the  here-and-now.  A  major  thrust  of  a  PM  will  always  be  to  invite  the 
customer  to  explore  a  larger  box  of  possibilities.  And  surely  it  is  true  that  no  customer  can 
develop  a  passion  for  your  product  without  envisioning  himself  and  his  part  of  the  battle 
somehow  transformed  by  it.  We  have  successfully  sold  training  and  tactical  systems  by 
repeatedly,  over  a  period  of  years,  exposing  users  to  virtual  simulations  of  them.  Education  leads 
to  realization.  Obviously,  very  little  of  this  is  quantifiable. .  .but  it  is  more  indispensable  than 
ever  today. 

Virtual  simulation’s  utility  as  a  communications  medium  designer  and  end  user  can  be 
extended  into  the  actual  design  phase  of  a  system.  By  using  end-users  to  help  design  the  soldier- 
machine  interface,  we  approach  standard  practices  in  the  commercial  software  industry: 
Preliminary  marketing  data,  alpha-  and  beta-test,  test  marketing,  etc.  Payoffs  are  again  rarely 
quantifiable  but  usually  undeniable:  a  better  product,  customer  buy-in,  and  ultimately  acceptance 
of  the  end-item.  Payoff  can  be  inferred,  however,  by  observing  rate  of  change  in  the  design 
originating  from  the  virtual  simulation  team.  Beware  of  changes  in  the  “test  market”  pool  of  end 
users;  high  turnover  rates  can  lead  to  opinionated,  circular  design  changes  rather  than  disciplined 
customer  feedback. 

Armor  tactics  (as  part  of  the  Blitzkreig )  were  not  developed  for  over  a  decade  after  the 
introduction  of  the  tank  into  warfare  -  and  by  the  other  side.  Why?  Most  people  cannot  or  will 
not  imagine  all  the  implications  of  new  hardware  and  capabilities.  But  virtual  simulations  offer 
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the  experience  without  the  equipment  -  and  the  opportunity  to  develop  new  tactics,  markets,  and 
applications.  Their  value  is  again  largely  unquantifiable  but  demonstrably  catalytic  in  both  time 
and  money  to  change  and  acceptance  of  new  systems. 

How  much  training  time  do  you  give  the  users  who  will  operate  your  system  during  DT  and 
OT?  Are  they  competent?  How  much  variation  and  operator  error  do  they  have?  How  will  you 
know?  Virtual  simulation  allows  the  PM  to  train  users  off-range,  screen  candidates  and 
determine  in  advance  the  performance  variation  expected  from  the  people.  It  can  also  discover 
design  flaws,  establish  expectations,  and  reduce  on-range  surprises.  The  test  community,  which 
often  has  no  experience  in  operating  or  measuring  the  new  system,  can  be  trained.  Every  hour  of 
training  time  that  can  be  avoided  on  the  range  is  a  direct  saving. 

The  idea  of  OT  in  a  virtual  environment  is  problematic,  but  offers  huge,  quantifiable 
savings  in  time,  risk,  and  money.  Virtual  simulation  is  not  likely  to  replace  OT;  only  a  zealot 
would  advocate  fielding  weapons  that  have  never  been  tried  in  “free  play.”  However,  in 
providing  a  basis  for  production  decisions,  virtual  simulation  offers  major  opportunities.  Where 
typical  DT/OT  missile  tests  are  focused  on  “does  the  missile  fly,”  while  having  only  a  few  rounds 
to  test,  virtual  simulation  can  answer  the  larger  question  “does  it  matter?” 

Similarly,  the  same  virtual  simulation  can  be  used  for  New  Equipment  Training  (NET) 
while  fielding  the  system.  By  taking  training  of  users  into  the  classroom  instead  of  the  range, 
savings  can  be  realized  in  targets  and  range  time.  Other  less  quantifiable  cost  savings  include 
increased  likelihood  of  acceptance,  more  positive  feedback,  and  a  more  data-based  (using 
recorded  data)  acceptance  process. 

The  huge  payoff  of  virtual  simulation,  especially  for  complex,  distributed  systems  such  as 
Command  And  Control  (C2)  -  lasting  for  decades  -  is  after  the  NET  team  has  left  and  the  unit 
begins  routine  training  and  operations.  Institutional  and  unit  training  systems  have  long  been 
recognized  as  money-savers.  Quantifiable  savings  should  be  expected  in  reduced  “live”  hours  to 
maintain  measured  proficiency  and  readiness  levels  in  the  unit.  Fewer  tactical  systems  (and 
spares)  should  be  allocated  to  the  “overhead”  of  institutional  training  -  a  direct,  large,  early 
saving  of  both  time  and  money  in  the  program. 

Then  the  big  phone  call  comes.  Load  up,  people,  and  head  to  Timbuktu.  How  to  stay 
sharp,  not  to  mention  rehearse  the  particular  mission,  while  your  unit’s  heavy  equipment  is 
loaded  nose-to-tail  in  the  bowels  of  a  ship?  The  warfighting  payoff  of  a  virtual  simulator  that  can 
“be”  anywhere,  any  time  of  year,  any  time  of  day,  any  weather  conditions  is  simply  incalculable. 
The  story  is  told  of  Ml  Abrams  tank  crews  training  shipboard  on  virtual  training  systems  below 
decks,  while  crossing  the  Atlantic  on  their  way  to  the  Persian  Gulf  during  Operation  Desert 
Shield.  That’s  the  way  to  go  into  battle.  At  the  time,  they  were  limited  to  school  training 
scenarios.  Today,  expect  a  mission-rehearsal  capability  to  result  from  a  virtual  simulator.  This 
implies  infrastructure  development  well  beyond  the  simulator  as  such:  rapid  scenario 
development,  worldwide  distribution,  and  a  host  of  other  issues.  Payoff  is  not  dollars,  but  lives. 
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Table  2.  Duration  and  Quantifiability  of  Virtual  Simulation  ROI  Throughout  a  System’s  Life 


Quantifiability  of  Simulation  ROI  Throughout  Phases  of  a  System's  Life 


Virtual  Simulation  Application 

Concept/ 
Tech.  Dev. 

Sys  Design/ 
Demo 

Concepts  and  Sales 

Low  ($) 

Design 

Low  (t,$) 

Low  (t,$) 

Employment/T  actics 

Low  (t,$) 

Med  (t,$) 

Test  design  &  Operator  Familiarization 

High  (t,$) 

Operational  Testing 

High  (t,$) 

New  Equipment  Training 
Skills  sustainment 
Mission  Rehearsal 


Production/ 

Deployment 


High  (t,$) 


Support/Operations 


Highjt,$) 

High  (t,  lives) 


VII.  ROI  OF  “STIMULATOR”  SIMULATION 

(simulated  people  and  environments,  real  equipment) 

Stimulator  simulation  is  defined  here  to  particularly  include  such  simulations  as  Hardware- 
in-the-Loop  (HWIL),  but  also  include  other  “real  thing”  simulators  such  as  subsystem  stimulators 
and  wind  tunnels.  A  major  use  of  such  systems  is  during  system  design;  use  of  various  subscale 
models  can  keep  designers  near  the  right  path.  Savings  are  difficult  to  estimate  -  how  do  you 
separate  costs  for  development  from  those  for  simulation?  Nonetheless,  a  program  absent  such 
simulation  support  is  conspicuous.  Stimulators’  descriptive  environments,  in  which  the  new 
system  can  be  exercised,  are  today  nearly  irreplaceable. 

At  a  slightly  higher  level  of  system  aggregation  -  say,  multiple  “black  boxes”  -  stimulators 
can  provide  considerable  payoff  in  pre-test  risk  reduction.  As  test  becomes  increasingly  integral 
to  design,  [4]  this  may  manifest  itself  in  much  problem-finding  and  fixing,  which  again  is 
difficult  to  quantify.  Additionally,  however,  these  tools  can  be  used  to  “pre-test”  the  system 
under  the  conditions  and  scenarios  expected  to  be  encountered  in  the  field.  The  phrase  “test  for 
success”  takes  on  real  meaning,  and  cost  savings  can  readily  be  quantified.  Serendipitously,  pre¬ 
testing  of  the  system  can  also  reveal  hidden  shortfalls  in  the  test  plans,  conditions,  and  apparatus 
-  before  range-time  bills  begin  to  accumulate. 

Stimulators  by  definition  involve  extensive  interfaces  with  the  real  equipment;  as  such,  they 
manifest  gathered  knowledge  about  the  system  and  the  ways  in  which  it  can  be  instrumented  and 
measured.  Qualification  and  field-test,  seeking  to  determine  similar  information,  can  often 
benefit  from  re-use  of  the  people  who  developed  the  stimulation  systems,  if  not  the  actual 
stimulator  instrumentation  outright.  Test  cost  savings  are  easily  quantified  using  actual  costs  for 
development  of  the  stimulator  instrumentation,  data  recording,  analysis  packages. 

As  in  the  virtual  OT  discussion  above,  the  Department  of  Defense  (DoD)  M&S  Master  Plan 
specifically  envisions  use  of  stimulators  for  regimes  of  test  and  evaluation  which  are  impossible 
in  that  they  pose  substantial  risk  to  human  life.  [6]  Wind-tunnel  and  HWIL  helicopter  flight- 
control  performance  assessments  at  and  beyond  the  “edge  of  the  envelope,”  for  example,  allow 
the  assessment  of  “impossible”  regimes  of  flight.  Even  where  human  life  is  not  at  risk, 
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stimulators  can  save  large  sums,  like  CAD,  in  avoidance  of  destructive  testing.  Savings  can  be 
quantified  as  they  are  proportional  to  range  time  avoidance  and  test  article  replacement  costs. 

Well-done  stimulation  paves  the  way  for  stimulator-based  acceptance  testing  -  units  can  be 
accepted  from  production  based  on  a  combination  of  performance  in  the  simulated  environment 
and  current  statistical  quality  control  data  from  the  production  line.  By  leveraging  already- 
available  data,  the  customer  can  avoid  paying  for  expensive,  redundant  flight  tests  for  lot 
acceptance.  This  payoff  is  particularly  marked  for  expendable  products  such  as  munitions,  and 
products  with  very  high  unit  costs. 

As  in  qualification,  field,  and  acceptance  testing,  the  same  stimulator  data  acquisition  and 
analysis  systems  can  be  applied  to  the  design  of  plant-floor  test  equipment.  The  industrial 
engineer  is  most  likely  to  find  the  most  knowledge  about  his  product’s  various  parts  among  the 
people  who  first  evaluated  (and  frequently  designed)  those  parts.  Drawings,  designs, 
instrumentation,  and  evaluation  methods  are  available  for  re-use  to  the  entrepreneurial  plant 
designer,  offering  highly  quantifiable  savings.  Again,  a  basis  of  estimate  exists  from  the  actual 
costs  incurred  during  initial  stimulator  buildup  -  and  replication  is  nearly  always  cheaper  than 
reinvention.  Savings  from  this  re-use  can  pay  off  throughout  the  life  of  the  system’s  production, 
through  simplified  configuration  control.  Further,  design  tolerances  can  be  loosened  or  tightened 
from  hands-on  experience.  Value  engineering  opportunities  abound,  as  stimulators  tend  to 
accumulate  large  quantities  of  system  performance  sensitivity  data. 

In-field  test  equipment,  while  generally  smaller  and  less  sophisticated  than  plant-floor 
production  equipment,  benefits  similarly  from  already-bought  knowledge.  Additionally,  if  end- 
product  designers  are  brought  together  with  the  stimulator  designers,  the  field-test  team  can 
leverage  all  the  lessons-leamed  on  how  the  product  tends  to  break  -  and  how  to  fix  it.  Like  plant 
test  equipment,  quantifiable  savings  accrue  throughout  production  and  deployment,  and  beyond 
throughout  the  system’s  entire  life. 

Hands-on  expertise  gained  from  stimulator  development  can  be  combined  with  reliability 
monitoring  and  modeling  to  reduce  ongoing  “does  it  still  work”  testing  through  a  product’s  life. 
Munitions  and  airframes,  for  example,  are  notoriously  subject  to  aging,  with  potentially 
catastrophic  outcomes.  Knowledge  gained  from  years  of  experience  in  finding  a  system’s 
tolerance  for  out-of-spec  operation  can  be  applied  directly  to  improving  constructive  reliability 
models  as  discussed  above,  with  reduced  requirements  for  additional  “fly  to  keep”  testing  and 
inspection  processes.  Quantifiable  savings  can  accrue  through  the  system’s  lifetime. 
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Table  3.  Duration  and  Quantifiability  of  “Stimulator”  Simulation  ROI 
Throughout  a  System’s  Life 

_ Quantifiability  of  Simulation  ROI  Throughout  Phases  of  a  System's  Life 

Concept/  Sys  Design/  Production/ 


"Stimulator"  Simulation  Application  Tech.  Dev. 

Demo 

Deployment 

Support/Operations 

Design 

Checkout/pre-test  risk  reduction 

Qualification/Field  Test  Instrumentation 

"Impossible"  Tests  (e.g.  lethal) 

Acceptance  Testing 

Plant-floor  Test  Equip.  Design 

Field  Automatic  Test  Equip.  Design 

Refine  RAM  fail/spare  models 

VIII.  ROI  OF  VIRTUAL/STIMULATOR  HYBRIDS 

(real  people  and  equipment,  simulated  environments) 

Hybrid  simulations  (in  this  definition)  offer  substantial  opportunities  for  savings  in  human- 
operated  systems.  They  seek  efficiencies  by  combining  both  man-in-the-loop  and  HWIL 
simulation,  often  in  the  same  lab  with  the  primary  products.  A  counter-example  is  instructive; 
consider  a  C2  system  whose  training  device  typically  led  the  end-product  in  development,  with 
100  percent  rewrite  of  code  for  the  end  product.  Wouldn’t  it  have  been  easier  and  cheaper  to  re¬ 
use  the  training  system  user  interfaces  in  the  end  product?  Wouldn’t  it  have  been  easier  to 
simply  use  the  end-product  processor  hardware  and  software  to  host  additional  simulator 
hardware/software  and  avoid  much  of  the  VV&A  associated  with  prescriptive  M&S  code? 

Hybrids  -  taking  many  forms  -  reap  many  of  the  benefits  of  both  virtual  and  stimulator 
simulation.  While  a  man-in-the-loop  simulator  might  cost  more  to  build  using  some  key  end 
product  components  such  as  flight  control  computers,  its  VV&A  costs  will  be  far  lower  -  the 
validation  of  “peripherals”  and  already-validated  environmental  models  can  be  far  simpler  (to  the 
extent  of  being  off-the-shelf,  in  some  cases)  than  a  whole-system  validation  effort. 
Fundamentally,  hybrids  seek  to  make  an  optimal  cost  tradeoff  between  buying/validating 
descriptive  models  of  “new”  components  and  simply  buying  the  components  themselves. 

Hybrids  offer  the  promise  of  high  levels  of  commonality  between  end  products,  training 
devices,  and  laboratory  testbeds,  developed  under  a  unified  team.  Savings  are  quantifiable 
during  development  in  that  as  costs  accrue  for  development  of  the  combined  effort,  some 
reasonable  estimates  for  a  more  traditional  “divided  house”  are  easily  obtainable.  Further, 
quantifiable  savings  can  continue  to  accrue  through  the  production  and  deployment  phase;  even 
the  operations  and  support  phase  of  the  product  life  can  realize  payoffs.  Consider  the  end 
product  and  its  support  products:  the  end-item,  its  training  devices,  its  development  lab,  and  its 
production/test  equipment  must  all  be  kept  in  synchronized  configuration  control.  (This  often 
means  keeping  several  versions  -  in-development,  in-production,  and  in-field,  for  example  - 
proximal  to  each  other.)  But  the  payoffs  are  very  large,  and  can  extend  well  beyond  even 
individual  product  lives. 
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Table  4.  Duration  and  Quantifiability  of  Hybrid  Virtual/Stimulator  Simulation 

Payoffs  Throughout  a  System’s  Life 

_ Quantifiability  of  Simulation  ROI  Throughout  Phases  of  a  System’s  Life 


Hybrid  Virtual/Stimulator  Simulation  Concept/  Sys  Design/  Production/ 


Tech.  Dev. 


Demo 


Design 

Test  design  &  Operator  Familiarization 

Checkout/pre-test  risk  reduction 

Operational  Testing 

"Impossible"  Tests  (e.g.  lethal) 

Employment/Tactics 

Product  Development  and  Support 

Support/Operations 


Med  (t,$) 


IX.  CONCLUSIONS 

Simulations  have  saved  many  customers  money  for  many  years.  Oftentimes,  we  have  failed 
to  fully  appreciate  or  recognize  the  magnitude  and  duration  of  those  savings.  Use  of  some  simple 
framework  for  allocating  not  only  simulation  costs,  but  savings,  could  help  our  industry  to  thrive 
in  an  increasingly-outsourced  world.  The  key  to  realizing  nearly  all  of  the  savings  identified  here 
is  the  intentional,  effective  transition  of  technology  and  information  amongst  the  components  and 
phases  of  a  system  development  program.  The  disciplines  required  to  accomplish  this  are  widely 
known,  but  strangely  under-applied.  By  developing  shared  teams,  language  and  technologies  in 
M&S  to  “narrow  the  bandwidth  of  problem  solving”  [2]  we  can  transform  simulation  from  a 
mere  technology  to  a  relevant,  viable  business  which  is  indispensable  to  our  customers  -  and 
known  by  them  to  be  so. 
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APPENDIX 

A  BUSINESS  CASE  FOR  MODELING  AND  SIMULATION 


Bio  -  20  years  with  the  Army 

Monte  Carlo  simulation  development  in  commo  and  radar  performance  analysis;  C2  systems 
design/sim 

Managed  weapon  system  product  improvements;  fire  control  systems  and  sensors,  controls  and 
displays 

Staff  for  missile  technology  development  and  transition 

DSMC  -  but  system  engineering  was  very  weak,  especially  in  simulation 

MS  IE  -  showed  me  where  I  had  succeeded  in  wasting  incredible  amounts  of  time  and  energy  in 

the  80’ s 


A-l 


What  I  Hope  Will  Happen 

(“Ah-ha!”) 

LEARNING 

•  Our  customers  see  things  differently  than  we  do:  cost,  schedule, 
performance 

•  We  can  provide  value  to  the  customer  by  controlling  the  variables  and 
finding  the  answers  to  his  questions  and  problems 

•  “Faith-based  simulation”  must  be  followed  by  products  the  customer  can  and 
will  appreciate 

•  We  have  a  good  story  here 

DECISION 

•  Tell  it  so  that  the  customer  can  understand  and  appreciate 

•  Leverage  each  others’  efforts  to  realize  larger  ROIs  than  ever 


This  business  of  simulation  is  not  new,  nor  is  the  economic  basis  of  its  use.  Major  General,  Doctor 
of  Engineering  Walter  Dornberger,  the  PM  for  the  German  A-4  rocket  program  (later,  better  known 
as  the  V-2)  spoke  for  every  PM  in  1942:  “the  makers  of  our  test  simulators  had  been  given  the  task 
of  substituting  simulators  for  time-consuming  experiments  carried  out  with  the  assembled  missile 
on  expensive  test  stands.”  Time,  money,  and  test  articles  -  the  3  bottlenecks  in  a  PM’s  life. 

(source:  V-2 ,  p  138,  Viking  Press,  1954) 

I  want  to  persuade  you  of  some  things.  I  also  want  you  to  make  some  decisions  based  on  what  we 
talk  about.  I  want  to  see  you  again  in  a  year  or  two,  changed,  and  more  successful  than  ever. ... 
Changed,  in  a  good  way. 

The  paper  is  where  the  information  is  -  I’m  hoping  to  convince  you  to  invest  your  time  reading  it, 
sifting  it,  and  applying  some  of  these  concepts  to  your  business. 

Per  Hank  Dubin,  dir  AMSO,  at  May  2001  SB  A  conference:  “. .  .the  most  difficult  part  is  learning 
how  to  measure  benefits.” 

I  also  hope  you  will  educate  me  about  your  business  model,  and  where  I  have  done  violence  to  it. 
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Cartoon  of  rocket  scientist  watching  missile  crash;  Army  officer  caption  says,  “12  years  and  85 
missions  dollars,  and  all  you  can  say  is  “D’OH!?” 

We’ve  done  this!  (in  half  the  time,  by  the  way. . .) 
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System/Product  Life  Phases 

(From  the  US  DoD  Project  Management  Model) 


-=  Swift  Decisive  Victory 


Concept  Component  Advanced 

Exploration  Development 

3aper  studies  of  Development  of  sub-  systems/ 

alternative  components  that  must  be 

concepts  for  demonstrated  before 

Sxit  criteria:  Concept/tech  demonstration  of 

Specific  concept  new  system  concepts 

to  be  pursued  &  Exit  criteria:  System  architecture 
technology  exists.  &  technology  mature. 


and  componeni 
Reduction  of  integratic 
Exit  criterion:  System 


System 

Demonstration 

Complete  development 

opment  models 
Combined  DT/OT 
Exit  criterion:  System 


Production  Rate  Prod  & 

|  Readiness/LRIP  Deployment 

&E,  LFT&E  of  Full  rate  production 
irod-rep  articles  Deployment  of  system 


This  is  the  life  of  a  project  manager,  at  least  in  the  US  DoD. 

More  and  more  managers  are  MBAs,  not  engineers.  Wall  Street  rules,  not  rocket  science.  The 
commoditization  of  technology  has  hurt  us;  the  mystique  is  gone  from  much  of  what  we  do.  The 
hardware  costs  have  shrunk  -  but  the  manpower  to  do  good  M&S  is  more  costly  than  ever,  at  least 
on  a  per-hour  basis. 

Duration  of  this  chart  -  you  need  to  know  which  part  of  this  timeline  your  customer  cares  about. 
“Support”  in  military  systems  may  be  up  to  30-50  years.  Also  in  public  works  -  dams,  roads, 
bridges.  But  in  telecomms,  may  be  around  1  year. 

In  my  world,  almost  all  of  our  M&S  business  is  in  the  first  2  phases,  tapering  down  in  third,  and 
very  little  in  4th. 

Again  -  your  value  is  constantly  held  up  to  this  yardstick.  Frequently,  your  product  is  information 
which  allows  the  project  manager  to  make  a  decision  about  which  way  to  go. .  .or  even  whether  to 
go  on  at  all.  Be  aware  of  HIS  decision  points,  and  his  departure  date. 
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What  do  we  aim  for? 

(more  bang  for  the  buck... euro... pound...) 


-=  Swift  Decisive  Victory 


S' 


COST  (e.g.  lives,  $,€,£  per  battle/decision,  life  cycle  cost) 


We  are  in  a  competitive  marketplace. 

Our  products  are  frequently  information,  prototypes,  designs,  test  equipment,  and  (gasp!)  opinions. 
Our  customers  make  decisions  -  influenced  by  us  -  about  the  relative  cost  effectiveness  of  our 
products  compared  to  other  alternatives. 

Our  job  is  to  develop  products  above  the  “slope”  defined  by  existing  products  in  the  market  - 
ideally,  in  the  green  areas. 
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Conceptual  Framework  of  Discussion 


_=  Swift  Decisive  Victory 


I  have  looked  at  a  fair  amount  of  literature  -  some  of  yours.  Everyone  picks  different  dimensions 
on  which  to  describe  the  domain  we  call  “modeling  and  simulation.”  I  wanted  a  way  to  subdivide 
the  domains  of  M&S  into  something  that  a  cost  analyst  or  a  project  manager  would  understand. 
Here  is  a  breakout  based  on  “how  much  reality”  an  evaluation  architecture  will  contain.  It 
conveniently  matches  expectations  of  many  cost  analysts  -  you  have  to  have  a  product,  it  has  to 
have  an  operator/user,  and  its  performs  within  some  sort  of  environmental  constraints  defined  by 
physics,  law,  culture,  or  some  other  boundary  condition. 
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•  Complex  System...  Complex  Simulation 

•  Test  Instrumentation  &  Calibration 

•  People... Human  Error 

•  “Weakest  Link”  effects 

‘One  more  thing  to  go  wrong.” 

-  an  old  ME’s  comment  on  a  turbocharger 


Regression  Model  (plot  data  first)  vs. 
Speculative  (draw  line  first) 

Descriptive  “should”  be  easier  to  make/buy 
“Reinvent  the  Wheel”  -  if  Necessary 
•Scope  and  Detail;  Range  of  Validity 
•End-Use(s)  and  Versatility 
•Resolution  in  Time,  Space,  etc. 


Availability  -  like  Reliability  combined  with  “how  to  fix  it”  time 

Some  of  the  best  advice  I  received  about  test  ranges  was,  make  a  pile  of  stuff  a  week  ahead  of  time, 
and  keep  adding  to  it  until  EVERYTHING  is  on  it.  Then  ship  it. 

“If  its  worth  taking  one  of  something. .  .its  worth  taking  two.” 

I  was  once  on  a  test  range  whose  tracking  system  would  be  used  to  cue  an  air  defense  system  -  but 
whose  accuracy  was  so  poor  that  it  cued  the  gunners  as  much  as  90  degrees  off  azimuth.  (A  good 
simulation  would  have  exhibited  no  more  than  5-10  degrees  bias.  Note  bias  worse  than  Normally 
distributed  errors.)  Worse,  no  one  had  mapped  where  it  experienced  those  biases.  It  was  a 
simulation. .  .an  incredibly  bad  one. 

People  are  not  only  “components”  of  the  test  system  architecture  -  they  are  highly  variable. 
Descriptive  vs.  Prescriptive  -  Credit  to  Paul  Fishwick  in  the  M&S  magazine  for  illustrating  two 
kinds  of  models. 

Prescriptive  suggests  a  possible  future;  descriptive  is  limited  to  the  present. 

The  easiest  curve  to  draw  is  the  one  with  the  least  data  -  fewer  degrees  of  statistical  freedom  gives 
more  latitude  for  artistic  freedom. 

Descriptive  is  fundamentally  different  -  able  to,  and  expected  to,  faithfully  represent  “the  real 
thing”.  Implications  in  make/buy,  VV&A,  and  emulation/simulation/HWIL 
Most  M&S  involves  a  prescriptive  system  operating  in  a  descriptive  environment 
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We  said  “cost-effectiveness”  was  important  to  customers. 

You  can  tell  me  about  “effectiveness.”  What  should  your  model  of  “cost”  look  like? 

What  kinds  of  simulation  are  available  to  solve  my  customers’  problems? 

Are  the  payoffs  quantifiable? 

How  high  is  the  ROI? 

Over  what  duration? 

How  long  must  my  faith  exceed  my  bills?  Hank  Dubin  (Dir,  AMSO)  talked  about  the  “faith 
factor”  at  the  SB  A  conference  in  May  2001. 

Every  PM  in  the  US  DoD  is  taught  about  earned  value  -  the  idea  that  cost  and  schedule  status  on  a 
project  can  be  denominated  in  dollars.  (Not  all  actually  learn  it  -  but  their  cost  analysts  know  it 
forwards  and  backwards.)  Most  don’t  experience  the  nice  graphics  here  -  they  just  get  cries  for 
help  when  variances  exceed,  say,  10%. 

Notice  the  “faith  factor”  after  you’ve  made  the  sale  -  but  are  still  developing  the  product.  The 
longer  this  is,  the  better  your  initial  sale  better  have  been. .  .lest  the  customer  decide  to  cut  his 
losses.  Remember,  in  his  schools  he  was  taught  not  to  make  decisions  on  the  basis  of  sunk 
cost. .  .only  future. 

Payoffs  are  great  -  but  in  a  world  subject  to  Moore’s  Law  (computing  capacity  of  microprocessors 
doubles  every  18  months)  our  customers  have  become  addicted  to  today’s  miracles  becoming 
tomorrow’s  doorstops. 

Your  golden  moment  to  sell  follow-on  work  is  after  the  ROI  goes  positive,  and  before  it  goes  flat. 
If  you  can  plan  your  miracles  to  occur  during  changes  of  customer,  so  much  the  better  -  you 
acquire  an  instant  reputation  as  a  miracle-worker. 

When  you  are  displacing  an  incumbent,  you  may  not  know  all  the  pitfalls  of  the  situation;  hence, 
Shakespeare’s  dictum. 
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The  unreadable  table  is  excerpted  from  the  paper;  here’s  some  eye-candy  to  illustrate  several  high- 
payoff  forms  of  simulation 

The  wireframe  CAD  model  is  obvious,  and  well-known.  This  particular  one  is  of  a  foreign  SAM 
system.  CAD  is  especially  nice  where  you  have  only  a  few  “real”  things  to  tinker  with.  These 
models  pay  off  for  a  long  time,  and,  unlike  bad  news,  improve  with  age. 

That  flowfield  model  around  a  CAD  drawing  (notice  the  linkage)  is  a  product  of  CFD.  Electrons 
versus  wind  tunnels,  themselves  a  form  of  simulation.  Remember  that  wind  tunnels  are  literally 
one-of-a-kind,  in  the  world.  “Fly”  on  your  schedule,  not  the  windows  left  around  everyone  else’s 
schedule.  Design  changes  are  CAD,  versus  machine  shop.  You  can  go  into  Mach  numbers 
unattainable  or  unaffordable  in  wind  tunnels.  Ditto  payoff,  especially  during  design. 

The  funny-colored  CAD-looking  model  is  a  radar  cross-section  model  -  think  of  survivability 
analyses  from  the  vehicle’s  point  of  view.  Turn  that  around,  and  this  would  be  part  of  a 
performance  analysis  for  a  sensor  system.  Especially  ditto  payoff  for  major  system-engineering 
problems  like  lifetime  RCS  reduction  and  control. 

The  pretty-colored  picture  is  a  thermal  model  of  a  Mach-5  missile  fin  about  2.5  seconds  after 
launch.  That  leading  edge  has  gone  from  ambient  air  temperature  to  around  2000  degrees  F.  The 
center  portion  is  around  600.  That  pod  has  RF  electronics  in  it,  with  skin  temperatures  around 
1000  degrees.  OK,  why  bother?  The  only  alternative  is  a  rocket  sled  test  track,  like  the  one  at 
Holloman  AFB  in  New  Mexico.  That  track  is  around  10  miles  long;  peak  speed  is  around  6000  fps 
for  recovery  (9000  for  non-recoverable).  Why  not  higher?  Things  come  off  the  track!  The 
workaround  is  a  3-mile-long  gasbag  filled  with  gas  of  different  density  that  air.  Argon  and  CO2, 
with  higher  density,  is  used  to  simulate  hypersonic  missile  flight  within  the  safety  limits  of  the 
track.  4000  fps  in  CO2  compares  for  aerothermal  effects  to  9500  ft/sec  in  air.  Helium  can  be  used 
to  simulate  other  effects  -  high  altitude,  faster  heat  rate,  etc.  (Other  facilities  are  blow-down,  laser 
ablation,  and  wind  tunnels)  Any  way  -  what  a  nightmare,  and  none  really  “complete.”  Hence, 
validated  thermal  modeling  is  of  great  utility  in  this  extreme  environment.  “Free”  compared  to  the 
alternative. 
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Process  simulation  is  old  hat  to  IEs,  but  not  always  recognized  in  the  larger  simulation  community. 
Nonetheless,  it  has  saved  a  fortune  in  diverse  applications  such  as  traffic  flow,  communications 
networks,  production  line  design,  supply  and  distribution  systems,  and  even  business  process  re¬ 
engineering  (here). 
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You  are  all  well  familiar  with  virtual  simulation  applied  to  training  systems;  here  are  3  variations 
which  saved  the  Army  a  lot  of  money,  and  are  still  saving  money. 

Gunners  went  from  1  in  5  to  5  in  5  performance.  This  has  a  quality  all  its  own  -  saved  lives, 
victory  with  fewer  casualties. 
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That  picture  is  of  a  helicopter-in-the-loop  -  around  60  feet  from  front  to  back.  Programmable 
hydraulics.  How  else  would  you  find  out  about  fatigue  effects  without  flying  it  and  finding  out  the 
hard  way?  Huge  cost  savings,  for  life. 

The  funny  looking  picture  is  the  nose  of  a  missile  -  radar  guided  in  this  case  -  that  thinks  its  flying 
toward  a  target.  The  stuff  around  it  is  anechoic  foam  to  produce  a  “free-space”  environment.  We 
have  surely  flown  millions  of  flights  this  way  by  now  -  we  literally  can’t  build  a  missile  without  it 
anymore.  This  way,  first  flights  can  actually  succeed!  Huge  cost  savings  in  design  and 
production/acceptance  test. 

The  Carco  table  -  the  thing  that  looks  like  a  gyro  in  its  gimbals  -  is  part  of  the  same  idea.  One  like 
this  was  recently  used  for  a  semi-active  laser  guided  missile.  First  flight  was  this 
spring. .  .successful.  How  do  you  measure  the  payoff  of  that? 
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Two  examples  of  an  extremely  valuable  hybrid  here  -  especially  so  for  things  that  fly,  like 
helicopters  and  missiles. 

Testbed  combines  real  fire-control  hardware  with  the  Avenger  training  system  and  some 
specialized  I/O  drivers.  Can  also  run  the  fire  control  as  software-in-the-loop  by  simply  changing 
the  I/O  file  name  declarations  in  the  front  of  the  program. 

Here,  fire  control  is  the  real  thing,  so  no  descriptive  simulation  needed. 

Operator  displays/controls  are  highly  descriptive  and  operator- validated. 

Simple,  modular  models  of  various  sensors  -  FLIR,  turret  stabilization  rates,  IFF,  etc. 
Environment  simulated. 

More  on  this  in  our  case  study  in  a  minute. 

Boeing  VITAL  (Vertical  Motion  Simulation/VMS  Integration  Technology  for  Affordable  Life 
Cycle  costs)  system  does  similar  thing  for  helicopters  -  man  in  the  loop,  simulated  flight  control 
functions,  real  aero.  (Here,  Apache.)  Big  design  phase  savings,  but  also  during  life  -  these  things 
are  out  there  for  30-40  years,  and  a  lot  of  modifications  get  made  in  that  time. 
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Overcoming  the  time  barrier 


-=  Swift  Decisive  Vit 


•  SHORAD  C2  Rgt/Spec 

•  Performance  Evaluation 

•  Weapon  interfaces 

•  ID  rule  development 


Expert  system  RaD 

FAAD  C2I A/B  specs,  PRISS, 

embedded  training 

System  performance  evaluation 


Performance  Assessment 

Test  and  Evaluation" 


P3I  Avenger  Rgt/Spec 
System  Perforr 

for  P3I,  GBSMS 


nger  ICOFT  *  — 


Training 


This  chart  illustrates  the  payoff  of  M&S  efforts  which  span  two  decades. 

No  substitute  for  a  graphical  representation  of  the  simulation.  If  you  haven’t  got  it,  get  it. 
Evolution  from  a  PDP  11-23  (ouch)  design  prototype  to  SGI  machines  used  in  virtual/stimulator 
hybrid. 

Funds  from  3  PMs  (ADCCS,  Stinger,  NLOS),  one  capital  equipment  fund,  2  tech  base  projects  (AI, 
LOGAD VISOR,  ID  sensor  fusion) 

Successful  M&S  must  transcend  PMs,  whose  only  observable  motivation  to  date  is  “on  my  watch.” 
Continuity  of  people  is  essential.  Preservation  of  intellectual  and  relationship  capital  was  primary 
enabler.  Necessary,  and  nearly  sufficient.  Time  is  enemy  number  two. 
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Watch  for  islands  of  simulation. .  .geographic  and  organizational. 
Many  barriers  and  disincentives  to  unity  of  effort  in  M&S 
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1994  effort 
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Simple  question  from  the  customer  -  important  to  keep  this  visible  to  everyone  in  the  project. 

Our  apparatus  involved  a  virtual/stimulator  hybrid,  using  real  soldiers  as  operators.  The  target 
environment  was  simulated  -  descriptive  model.  Descriptive  model  of  radars  and  cueing  system. 
Descriptive/HWIL  stimulator  model  of  Avenger  cockpit  and  fire  control.  Prescriptive 
model/prototype  of  operator  display.  Linkage  between  radar  and  gunner  prescriptive/hypothesized. 
Short  schedule  -  support  acquisition  decision  for  GBS  -  now  Sentinel. 

Verify  worth  with  a  field  test,  replicating  the  scenarios. 

Validate  assessment  -  not  full  model  validation. 
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Simple  apparatus  -  high  reliability  of  hardware  and  software. 
Descriptive  weapon  and  radar  models;  Prescriptive  C2  cueing  model. 

1994  case  study 


Is  it  real  or  is  it  Memorex?  This  happens  to  be  real  hardware  -  the  only  way  I  can  tell  it’s  the  real 
thing  isn’t  the  control  panel,  it’s  the  fire  extinguisher.  In  the  simulator,  all  the  hardware  looks  like 
this. 

The  left  display/control  group  is  the  item  being  evaluated. 
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Counter-UAV  Simulation  Findings 


Percent  of  Targets  Killed  by  SHORAD  Fire  Unit  Gunners 

C2I  Manual 

C2I  Automatic 

Scenario 

1 

2 

1  3 

1  4 

1  1  2  | 

1  3  | 

1  4 

la 

0.38 

0.63 

0.13 

0.25 

0.50  0.50 

0.63 

0.25 

1b 

0.38 

0.38 

0.38 

0.25 

0.38  0.38 

0.63 

0.38 

1c 

0.13 

0.38 

0.38 

0.25 

0.38  0.38 

0.38 

0.13 

2 

0.54 

0.69 

0.46 

0.77 

0.85 

0.69 

3 

0.50 

0.44 

0.38 

0.72 

1.00 

0.67 

4 

0.42 

0.58 

0.25 

0.58 

0.83 

0.33 

5 

0.30 

0.70 

0.40 

0.30 

0.50  0.70 

1.00 

0.60 

6 

0.70 

0.70 

0.60 

0.10 

1 .00  1 .00 

1.00 

1.00 

Missing  Data; 
not  analyzed 


Percent  Kills  by  C2I  Type 


What  the  customer 


was  hoping  to  hear: 
Cueing  improves 
performance. 


But  that  isn’t  the  whole  story... 


Three  dimensions  of  interest  here  (known  sources  of  variation). 

Scenario 

Operator 

Type  of  target  cueing  -  automated  system  just  shown,  or  another,  semi-automated  “manual”  system 
with  a  voice-tell  in  the  process. 

We  found  the  answer  to  the  question  -  the  new  system  was  a  quantum  improvement  over  the  old. 
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More  Effects  than  Anticipated 

(but  this  is  where  M&S  shines) 


Problem:  Two  nuisance  variables  can  invert  effects  of  C2  system: 


-=  Swift  Decisive  Victory 


People/Operator  Effects  Cueing  System  Effects  Environment/Scenario  Effects 


Control  over  nuisance  variables  is  essential  to  avoid  masking  effects  of  interest. 
1994  case  study  It  is  also  rare. 


BUT 

Mostly,  we  got  lucky.  Here’s  a  comparison  of  the  relative  magnitude  of  the  three  dimensions  of 
the  scenario,  using  bar-and-whisker  charts.  All  have  “number  of  kills”  as  their  vertical  axis,  scaled 
the  same. 

Variation  is  larger  from  the  two  variables  we  thought  we  had  control  of. 

4  different  operators  -  #4  a  loser. 

5  different  scenarios  -  #5  like  shooting  fish  in  a  barrel,  apparently.  Tends  to  wash  out  differences 
we’re  looking  for. 

The  good  news  (other  than  a  validated  hypothesis):  The  field  test  exercise  was  scoped  to  reduce 
some  of  this,  the  results  were  consistent,  the  GBS  was  bought,  the  C2I  system  was  bought,  the  fire 
control  was  bought. .  .and  today  this  stuff  is  in  the  field. 

“Real  simulation”  can  put  a  drive  in  a  program  that  nothing  else  can. 
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Moral  of  the  Story 


(and  the  paper) 


-=  Swift  Decisive  Victory 


Costs  could  have  been  higher...  much... 

Success  could  have  been  avoided...  easily... 

Schedule  could  have  been  stretched...  to  “next  year”... 
Test/analyst/customer  teamwork  could  have  been  prevented... 
Travel  to  “exotic”  test  ranges  could  have  been  greater... 

...if  only  we  hadn’t  used  that  doggone  simulation! 


‘A  closed  mouth  gathers  no  feet.” 
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Tying  it  Together 

(A  10-Step  Program) 


1.  Define  the  simulation’s  product  and  its  customer. 

2.  Identify  customers  for  specific  deliverables  (paper,  demo,  hardware, 
evaluation,  etc.). 

3.  When  does  the  customer  need  the  simulation? 

4.  How  mature  are  the  M&S  components,  relative  to  what  the  customer  is  able 
and  willing  to  accept,  and  when  he  wants  it? 

5.  Make  or  buy?  Is  maturation  of  the  M&S  components  adaptive, 
evolutionary,  or  revolutionary?  Are  there  other  constraints  (legal, 
monetary,  etc.)? 

6.  Just  how  important  is  this  technology  and  customer  need  compared  to  all 
others? 

7.  Who  in  our  company  will  lead  the  program?  Can  they  amass  sufficient 
intellectual  and  relationship  capital  to  succeed? 

8.  Can  a  single  simulation  serve  several  customers,  together  or  in  stages?  Make 
or  buy? 

9.  What  are  interim  milestones,  incentives,  metrics,  and  exit  criteria? 

10.  How,  and  to  whom,  will  we  transition  evaluations  and  other  deliverables  into 
the  customers’  decision  processes? 
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